Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center
نویسندگان
چکیده
We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI 1. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated schedulers of today’s frameworks, Mesos introduces a distributed two-level scheduling mechanism called resource offers. Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. Our experimental results show that Mesos can achieve near-optimal locality when sharing the cluster among diverse frameworks, can scale up to 50,000 nodes, and is resilient to node failures.
منابع مشابه
Performance Interference of Multi-tenant, Big Data Frameworks in Resource Constrained Private Clouds
In this paper, we investigate and characterize the behavior of “big” and “fast” data analysis frameworks, in multitenant, shared settings for which computing resources (CPU and memory) are limited. Such settings and frameworks are frequently employed in both public and private cloud deployments. Resource constraints stem from both physical limitations (private clouds) and what the user is willi...
متن کاملMROrchestrator: A Fine-Grained Resource Orchestration Framework for Hadoop MapReduce
Efficient resource management in data centers and clouds running large distributed data processing frameworks like Hadoop is crucial for enhancing the performance of hosted MapReduce applications, and boosting the resource utilization. However, existing resource scheduling schemes in Hadoop allocate resources at the granularity of fixed-size, static portions of the nodes, called slots. A slot r...
متن کاملMegos: Enterprise Resource Management in Mesos Clusters
Enterprise data centers increasingly adopt a cloud-like architecture that enables the execution of multiple workloads on a shared pool of resources, reduces the data center footprint and drives down the costs. The Apache Mesos project is emerging as a leading open source resource management technology for server clusters. However, the default Mesos allocation mechanism lacks a number of policy ...
متن کاملاندازهگیری میزان صرفهجوییهای زمانی حاصل از مشترکسازی مولفهها در زنجیرههای تأمین
Sharing common resources is amongst critical factors creating competitive advantages in business and manufacturing. In today’s competitive and dynamic environment, application of the resource sharing approach has become the focal point of attention for business managers. By resource sharing and through common platform guidelines, the possibility of producing an extended variety of products us...
متن کاملResource Revocation in Apache Mesos
We demonstrate how adding resource revocation to Mesos allows the system to provide latency and resource guarantees to frameworks. Mesos, which uses dominant resource fairness to offers of new resources, was designed initially for primarly MapReduce-like and found to provide weak guarantees with more general workloads. This project resolved this issue by allowing frameworks to explicitely state...
متن کامل